Face model capture by a wearable device

ABSTRACT

Systems and methods for generating a face model for a user of a head-mounted device are disclosed. The head-mounted device can include one or more eye cameras configured to image the face of the user while the user is putting the device on or taking the device off. The images obtained by the eye cameras may be analyzed using a stereoscopic vision technique, a monocular vision technique, or a combination, to generate a face model for the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority under 35 U.S.C. § 119(e)to U.S. Provisional Application No. 62/400,907, filed on Sep. 28, 2016,entitled “FACE MODEL CAPTURE BY AN AUGMENTED REALITY DEVICE,” thedisclosure of which is hereby incorporated by reference herein in itsentirety.

FIELD

The present disclosure relates to virtual reality and augmented realityimaging and visualization systems and more particularly to generating aface model of a user of such systems.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality”, “augmentedreality”, or “mixed reality” experiences, wherein digitally reproducedimages or portions thereof are presented to a user in a manner whereinthey seem to be, or may be perceived as, real. A virtual reality, or“VR”, scenario typically involves presentation of digital or virtualimage information without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user; a mixed reality,or “MR”, related to merging real and virtual worlds to produce newenvironments where physical and virtual objects co-exist and interact inreal time. As it turns out, the human visual perception system is verycomplex, and producing a VR, AR, or MR technology that facilitates acomfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements ischallenging. Systems and methods disclosed herein address variouschallenges related to VR, AR and MR technology.

SUMMARY

Various embodiments of a mixed reality system for capturing face imagesand determining a face model are disclosed.

Systems and methods for generating a face model for a user of ahead-mounted device are disclosed. The head-mounted device can includeone or more eye cameras configured to image the face of the user whilethe user is putting the device on or taking the device off. The imagesobtained by the eye cameras may be analyzed using a stereoscopic visiontechnique, a monocular vision technique, or a combination, to generate aface model for the user.

Details of one or more implementations of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims. Neitherthis summary nor the following detailed description purports to defineor limit the scope of the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson.

FIG. 2 schematically illustrates an example of a wearable system.

FIG. 3 schematically illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes.

FIG. 4 schematically illustrates an example of a waveguide stack foroutputting image information to a user.

FIG. 5 shows example exit beams that may be outputted by a waveguide.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem, usedin the generation of a multi-focal volumetric display, image, or lightfield.

FIG. 7 is a block diagram of an example of a wearable system.

FIG. 8 is a process flow diagram of an example of a method of renderingvirtual content in relation to recognized objects.

FIG. 9 is a block diagram of another example of a wearable system.

FIG. 10 is a process flow diagram of an example of a method forinteracting with a virtual user interface.

FIG. 11 illustrates an example wearable device which can acquire imagesof a user's face while the user is putting on (or taking off) thewearable device.

FIG. 12 illustrates an example process for generating a face model.

FIG. 13A describes an example process of generating a face model usingstereo vision techniques.

FIG. 13B describes an example process of generating a face model usingmonocular vision techniques.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

DETAILED DESCRIPTION

Overview

A user of an augmented or a virtual reality system can use a wearabledevice, such as a head mounted display (HMD) to immerse in analternative world with virtual objects. Sometimes, the wearable devicemay present an avatar (which includes, e.g., a virtual image) of theuser in that alternative world for interactions with other users. Toprovide realistic images and movements for the avatar, the wearabledevice can provide the avatar images based on the user's facial look andexpressions. The avatar image may be built based on the images acquiredby one or more imaging systems of the wearable device. The imagingsystems can include an inward-facing imaging system which can compriseeye cameras to track user's eye movements and an outward-facing imagingsystem which can comprise cameras for imaging the user's environment.However, the imaging systems of the wearable device cannot easily imagethe face of the user once it is placed on the user's head. For example,the inward-facing imaging system can be configured to image theperiocular region of the user when the wearable device worn by the userand the eye cameras may not have a large enough field of view forimaging the user's whole face. As another example, the cameras of theoutward-facing imaging system are configured to point away from the userwhen the user wears the wearable device and thus cannot easily obtain aface image of the user. This results in a variety of difficulties forgenerating an acceptable image for rendering the virtual avatar.

The wearable device described herein is directed to reducing thesedifficulties by providing an imaging system configured to obtain imagesof the user's face while the user is putting on or taking off thewearable device. Advantageously, the wearable device can use theinward-facing imaging system to obtain images of the user's face whilethe user is putting on or taking off the device, which provides anunconventional application of the inward-facing imaging system (whosepurpose is eye tracking) to acquire face images. Further, the wearabledevice can automatically start and stop imaging the user's face bydetecting a starting or a stopping trigger (e.g., which may be based onthe images acquired by the wearable device or based on the movement ofthe wearable device). Advantageously, by automatically acquiring imageswhile the user is putting on or taking off the device, the user may notneed to perform additional actions (e.g., rotating or moving thewearable device around the user's head) in order for the wearable deviceto generate a face model. Also, by stopping imaging when the wearabledevice is seated on the user's face, the inward-facing imaging systemcan automatically begin its (typically) primary function of tracking theuser's eyes.

The images can include still images, photographs, animations, individualframes from a video, or a video. The wearable device may build athree-dimensional (3D) model of the user's face based on the imagesacquired by the imaging system. For example, the wearable device canhave two eye cameras each configured to video a region of the user'sface. For each frame of the video, the wearable device can synthesizeimages acquired by the two eye cameras to generate the 3D face model.Additionally or alternatively, the wearable device can separatelysynthesize images acquired by each eye camera and combine thesynthesized the images for each eye camera to generate the 3D facemodel.

The resulting model may be used for purposes such as generating avirtual avatar, determining fit of the wearable device, performing useridentification, performing image registration, or tuning operationalparameters of the wearable device such as, for example, adjusting therendering locations of the virtual images, the relative position ororientation of the light projectors, etc., based on the interocularseparation of the user's eyes (e.g., an inter-pupillary distance) orother metric of the user's face

Examples of 3D Display of a Wearable System

A wearable system (also referred to herein as an augmented reality (AR)system) can be configured to present 2D or 3D virtual images to a user.The images may be still images, frames of a video, or a video, incombination or the like. At least a portion of the wearable system canbe implemented on a wearable device that can present a VR, AR, or MRenvironment, alone or in combination, for user interaction. The wearabledevice can be a head-mounted device (HMD) which is used interchangeablyas an AR device (ARD). Further, for the purpose of the presentdisclosure, the term “AR” is used interchangeably with the term “MR”.

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson. In FIG. 1, an MR scene 100 is depicted wherein a user of an MRtechnology sees a real-world park-like setting 110 featuring people,trees, buildings in the background, and a concrete platform 120. Inaddition to these items, the user of the MR technology also perceivesthat he “sees” a robot statue 130 standing upon the real-world platform120, and a cartoon-like avatar character 140 flying by which seems to bea personification of a bumble bee, even though these elements do notexist in the real world.

In order for the 3D display to produce a true sensation of depth, andmore specifically, a simulated sensation of surface depth, it may bedesirable for each point in the display's visual field to generate anaccommodative response corresponding to its virtual depth. If theaccommodative response to a display point does not correspond to thevirtual depth of that point, as determined by the binocular depth cuesof convergence and stereopsis, the human eye may experience anaccommodation conflict, resulting in unstable imaging, harmful eyestrain, headaches, and, in the absence of accommodation information,almost a complete lack of surface depth.

VR, AR, and MR experiences can be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer's eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane or based on observing differentimage features on different depth planes being out of focus. Asdiscussed elsewhere herein, such depth cues provide credible perceptionsof depth.

FIG. 2 illustrates an example of wearable system 200 which can beconfigured to provide an AR/VR/MR scene. The wearable system 200 canalso be referred to as the AR system 200. The wearable system 200includes a display 220, and various mechanical and electronic modulesand systems to support the functioning of display 220. The display 220may be coupled to a frame 230, which is wearable by a user, wearer, orviewer 210. The display 220 can be positioned in front of the eyes ofthe user 210. The display 220 can present AR/VR/MR content to a user.The display 220 can comprise a head mounted display that is worn on thehead of the user.

In some embodiments, a speaker 240 is coupled to the frame 230 andpositioned adjacent the ear canal of the user (in some embodiments,another speaker, not shown, is positioned adjacent the other ear canalof the user to provide for stereo/shapeable sound control). The display220 can include an audio sensor (e.g., a microphone) 232 for detectingan audio stream from the environment and capture ambient sound. In someembodiments, one or more other audio sensors, not shown, are positionedto provide stereo sound reception. Stereo sound reception can be used todetermine the location of a sound source. The wearable system 200 canperform voice or speech recognition on the audio stream.

The wearable system 200 can include an outward-facing imaging system 464(shown in FIG. 4) which observes the world in the environment around theuser. The wearable system 200 can also include an inward-facing imagingsystem 462 (shown in FIG. 4) which can track the eye movements of theuser. The inward-facing imaging system may track either one eye'smovements or both eyes' movements. The inward-facing imaging system 462may be attached to the frame 230 and may be in electrical communicationwith the processing modules 260 or 270, which may process imageinformation acquired by the inward-facing imaging system to determine,e.g., the pupil diameters or orientations of the eyes, eye movements oreye pose of the user 210. The inward-facing imaging system 462 mayinclude one or more cameras. For example, at least one camera may beused to image each eye. The images acquired by the cameras may be usedto determine pupil size or eye pose for each eye separately, therebyallowing presentation of image information to each eye to be dynamicallytailored to that eye. As another example, the pupil diameter ororientation of only one eye is determined (e.g., based on imagesacquired for a camera configured to acquire the images of that eye) andthe eye features determined for this eye are assumed to be similar forthe other eye of the user 210.

As an example, the wearable system 200 can use the outward-facingimaging system 464 or the inward-facing imaging system 462 to acquireimages of a pose of the user. The images may be still images, frames ofa video, or a video.

The display 220 can be operatively coupled 250, such as by a wired leador wireless connectivity, to a local data processing module 260 whichmay be mounted in a variety of configurations, such as fixedly attachedto the frame 230, fixedly attached to a helmet or hat worn by the user,embedded in headphones, or otherwise removably attached to the user 210(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local processing and data module 260 may comprise a hardwareprocessor, as well as digital memory, such as non-volatile memory (e.g.,flash memory), both of which may be utilized to assist in theprocessing, caching, and storage of data. The data may include data a)captured from sensors (which may be, e.g., operatively coupled to theframe 230 or otherwise attached to the user 210), such as image capturedevices (e.g., cameras in the inward-facing imaging system or theoutward-facing imaging system), audio sensors (e.g., microphones),inertial measurement units (IMUs), accelerometers, compasses, globalpositioning system (GPS) units, radio devices, or gyroscopes; or b)acquired or processed using remote processing module 270 or remote datarepository 280, possibly for passage to the display 220 after suchprocessing or retrieval. The local processing and data module 260 may beoperatively coupled by communication links 262 or 264, such as via wiredor wireless communication links, to the remote processing module 270 orremote data repository 280 such that these remote modules are availableas resources to the local processing and data module 260. In addition,remote processing module 280 and remote data repository 280 may beoperatively coupled to each other.

In some embodiments, the remote processing module 270 may comprise oneor more processors configured to analyze and process data or imageinformation. In some embodiments, the remote data repository 280 maycomprise a digital data storage facility, which may be available throughthe internet or other networking configuration in a “cloud” resourceconfiguration. In some embodiments, all data is stored and allcomputations are performed in the local processing and data module,allowing fully autonomous use from a remote module.

The human visual system is complicated and providing a realisticperception of depth is challenging. Without being limited by theory, itis believed that viewers of an object may perceive the object as beingthree-dimensional due to a combination of vergence and accommodation.Vergence movements (i.e., rolling movements of the pupils toward or awayfrom each other to converge the lines of sight of the eyes to fixateupon an object) of the two eyes relative to each other are closelyassociated with focusing (or “accommodation”) of the lenses of the eyes.Under normal conditions, changing the focus of the lenses of the eyes,or accommodating the eyes, to change focus from one object to anotherobject at a different distance will automatically cause a matchingchange in vergence to the same distance, under a relationship known asthe “accommodation-vergence reflex.” Likewise, a change in vergence willtrigger a matching change in accommodation, under normal conditions.Display systems that provide a better match between accommodation andvergence may form more realistic and comfortable simulations ofthree-dimensional imagery.

FIG. 3 illustrates aspects of an approach for simulating athree-dimensional imagery using multiple depth planes. With reference toFIG. 3, objects at various distances from eyes 302 and 304 on the z-axisare accommodated by the eyes 302 and 304 so that those objects are infocus. The eyes 302 and 304 assume particular accommodated states tobring into focus objects at different distances along the z-axis.Consequently, a particular accommodated state may be said to beassociated with a particular one of depth planes 306, which has anassociated focal distance, such that objects or parts of objects in aparticular depth plane are in focus when the eye is in the accommodatedstate for that depth plane. In some embodiments, three-dimensionalimagery may be simulated by providing different presentations of animage for each of the eyes 302 and 304, and also by providing differentpresentations of the image corresponding to each of the depth planes.While shown as being separate for clarity of illustration, it will beappreciated that the fields of view of the eyes 302 and 304 may overlap,for example, as distance along the z-axis increases. In addition, whileshown as flat for the ease of illustration, it will be appreciated thatthe contours of a depth plane may be curved in physical space, such thatall features in a depth plane are in focus with the eye in a particularaccommodated state. Without being limited by theory, it is believed thatthe human eye typically can interpret a finite number of depth planes toprovide depth perception. Consequently, a highly believable simulationof perceived depth may be achieved by providing, to the eye, differentpresentations of an image corresponding to each of these limited numberof depth planes.

Waveguide Stack Assembly

FIG. 4 illustrates an example of a waveguide stack for outputting imageinformation to a user. A wearable system 400 includes a stack ofwaveguides, or stacked waveguide assembly 480 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 432 b, 434 b, 436 b, 438 b, 4400 b. In some embodiments,the wearable system 400 may correspond to wearable system 200 of FIG. 2,with FIG. 4 schematically showing some parts of that wearable system 200in greater detail. For example, in some embodiments, the waveguideassembly 480 may be integrated into the display 220 of FIG. 2.

With continued reference to FIG. 4, the waveguide assembly 480 may alsoinclude a plurality of features 458, 456, 454, 452 between thewaveguides. In some embodiments, the features 458, 456, 454, 452 may belenses. In other embodiments, the features 458, 456, 454, 452 may not belenses. Rather, they may simply be spacers (e.g., cladding layers orstructures for forming air gaps).

The waveguides 432 b, 434 b, 436 b, 438 b, 440 b or the plurality oflenses 458, 456, 454, 452 may be configured to send image information tothe eye with various levels of wavefront curvature or light raydivergence. Each waveguide level may be associated with a particulardepth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 420, 422,424, 426, 428 may be utilized to inject image information into thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b, each of which may beconfigured to distribute incoming light across each respectivewaveguide, for output toward the eye 410 (which may correspond to theeye 304 in FIG. 3). Light exits an output surface of the image injectiondevices 420, 422, 424, 426, 428 and is injected into a correspondinginput edge of the waveguides 440 b, 438 b, 436 b, 434 b, 432 b. In someembodiments, a single beam of light (e.g., a collimated beam) may beinjected into each waveguide to output an entire field of clonedcollimated beams that are directed toward the eye 410 at particularangles (and amounts of divergence) corresponding to the depth planeassociated with a particular waveguide.

In some embodiments, the image injection devices 420, 422, 424, 426, 428are discrete displays that each produce image information for injectioninto a corresponding waveguide 440 b, 438 b, 436 b, 434 b, 432 b,respectively. In some other embodiments, the image injection devices420, 422, 424, 426, 428 are the output ends of a single multiplexeddisplay which may, e.g., pipe image information via one or more opticalconduits (such as fiber optic cables) to each of the image injectiondevices 420, 422, 424, 426, 428.

A controller 460 controls the operation of the stacked waveguideassembly 480 and the image injection devices 420, 422, 424, 426, 428.The controller 460 includes programming (e.g., instructions in anon-transitory computer-readable medium) that regulates the timing andprovision of image information to the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, the controller 460 may be a singleintegral device, or a distributed system connected by wired or wirelesscommunication channels. The controller 460 may be part of the processingmodules 260 or 270 (illustrated in FIG. 2) in some embodiments.

The waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be configured topropagate light within each respective waveguide by total internalreflection (TIR). The waveguides 440 b, 438 b, 436 b, 434 b, 432 b mayeach be planar or have another shape (e.g., curved), with major top andbottom surfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 440 b, 438 b,436 b, 434 b, 432 b may each include light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 410. Extracted light may also be referred to as outcoupledlight, and light extracting optical elements may also be referred to asoutcoupling optical elements. An extracted beam of light is outputted bythe waveguide at locations at which the light propagating in thewaveguide strikes a light redirecting element. The light extractingoptical elements (440 a, 438 a, 436 a, 434 a, 432 a) may, for example,be reflective or diffractive optical features. While illustrateddisposed at the bottom major surfaces of the waveguides 440 b, 438 b,436 b, 434 b, 432 b for ease of description and drawing clarity, in someembodiments, the light extracting optical elements 440 a, 438 a, 436 a,434 a, 432 a may be disposed at the top or bottom major surfaces, or maybe disposed directly in the volume of the waveguides 440 b, 438 b, 436b, 434 b, 432 b. In some embodiments, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be formed in a layer ofmaterial that is attached to a transparent substrate to form thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b. In some other embodiments,the waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be a monolithicpiece of material and the light extracting optical elements 440 a, 438a, 436 a, 434 a, 432 a may be formed on a surface or in the interior ofthat piece of material.

With continued reference to FIG. 4, as discussed herein, each waveguide440 b, 438 b, 436 b, 434 b, 432 b is configured to output light to forman image corresponding to a particular depth plane. For example, thewaveguide 432 b nearest the eye may be configured to deliver collimatedlight, as injected into such waveguide 432 b, to the eye 410. Thecollimated light may be representative of the optical infinity focalplane. The next waveguide up 434 b may be configured to send outcollimated light which passes through the first lens 452 (e.g., anegative lens) before it can reach the eye 410. First lens 452 may beconfigured to create a slight convex wavefront curvature so that theeye/brain interprets light coming from that next waveguide up 434 b ascoming from a first focal plane closer inward toward the eye 410 fromoptical infinity. Similarly, the third up waveguide 436 b passes itsoutput light through both the first lens 452 and second lens 454 beforereaching the eye 410. The combined optical power of the first and secondlenses 452 and 454 may be configured to create another incrementalamount of wavefront curvature so that the eye/brain interprets lightcoming from the third waveguide 436 b as coming from a second focalplane that is even closer inward toward the person from optical infinitythan was light from the next waveguide up 434 b.

The other waveguide layers (e.g., waveguides 438 b, 440 b) and lenses(e.g., lenses 456, 458) are similarly configured, with the highestwaveguide 440 b in the stack sending its output through all of thelenses between it and the eye for an aggregate focal powerrepresentative of the closest focal plane to the person. To compensatefor the stack of lenses 458, 456, 454, 452 when viewing/interpretinglight coming from the world 470 on the other side of the stackedwaveguide assembly 480, a compensating lens layer 430 may be disposed atthe top of the stack to compensate for the aggregate power of the lensstack 458, 456, 454, 452 below. Such a configuration provides as manyperceived focal planes as there are available waveguide/lens pairings.Both the light extracting optical elements of the waveguides and thefocusing aspects of the lenses may be static (e.g., not dynamic orelectro-active). In some alternative embodiments, either or both may bedynamic using electro-active features.

With continued reference to FIG. 4, the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be configured to bothredirect light out of their respective waveguides and to output thislight with the appropriate amount of divergence or collimation for aparticular depth plane associated with the waveguide. As a result,waveguides having different associated depth planes may have differentconfigurations of light extracting optical elements, which output lightwith a different amount of divergence depending on the associated depthplane. In some embodiments, as discussed herein, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be volumetric orsurface features, which may be configured to output light at specificangles. For example, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a may be volume holograms, surface holograms, and/ordiffraction gratings. Light extracting optical elements, such asdiffraction gratings, are described in U.S. Patent Publication No.2015/0178939, published Jun. 25, 2015, which is incorporated byreference herein in its entirety.

In some embodiments, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a are diffractive features that form a diffractionpattern, or “diffractive optical element” (also referred to herein as a“DOE”). Preferably, the DOE has a relatively low diffraction efficiencyso that only a portion of the light of the beam is deflected away towardthe eye 410 with each intersection of the DOE, while the rest continuesto move through a waveguide via total internal reflection. The lightcarrying the image information can thus be divided into a number ofrelated exit beams that exit the waveguide at a multiplicity oflocations and the result is a fairly uniform pattern of exit emissiontoward the eye 304 for this particular collimated beam bouncing aroundwithin a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”state in which they actively diffract, and “off” state in which they donot significantly diffract. For instance, a switchable DOE may comprisea layer of polymer dispersed liquid crystal, in which microdropletscomprise a diffraction pattern in a host medium, and the refractiveindex of the microdroplets can be switched to substantially match therefractive index of the host material (in which case the pattern doesnot appreciably diffract incident light) or the microdroplet can beswitched to an index that does not match that of the host medium (inwhich case the pattern actively diffracts incident light).

In some embodiments, the number and distribution of depth planes ordepth of field may be varied dynamically based on the pupil sizes ororientations of the eyes of the viewer. Depth of field may changeinversely with a viewer's pupil size. As a result, as the sizes of thepupils of the viewer's eyes decrease, the depth of field increases suchthat one plane that is not discernible because the location of thatplane is beyond the depth of focus of the eye may become discernible andappear more in focus with reduction of pupil size and commensurate withthe increase in depth of field. Likewise, the number of spaced apartdepth planes used to present different images to the viewer may bedecreased with the decreased pupil size. For example, a viewer may notbe able to clearly perceive the details of both a first depth plane anda second depth plane at one pupil size without adjusting theaccommodation of the eye away from one depth plane and to the otherdepth plane. These two depth planes may, however, be sufficiently infocus at the same time to the user at another pupil size withoutchanging accommodation.

In some embodiments, the display system may vary the number ofwaveguides receiving image information based upon determinations ofpupil size or orientation, or upon receiving electrical signalsindicative of particular pupil size or orientation. For example, if theuser's eyes are unable to distinguish between two depth planesassociated with two waveguides, then the controller 460 (which may be anembodiment of the local processing and data module 260) can beconfigured or programmed to cease providing image information to one ofthese waveguides. Advantageously, this may reduce the processing burdenon the system, thereby increasing the responsiveness of the system. Inembodiments in which the DOEs for a waveguide are switchable between theon and off states, the DOEs may be switched to the off state when thewaveguide does receive image information.

In some embodiments, it may be desirable to have an exit beam meet thecondition of having a diameter that is less than the diameter of the eyeof a viewer. However, meeting this condition may be challenging in viewof the variability in size of the viewer's pupils. In some embodiments,this condition is met over a wide range of pupil sizes by varying thesize of the exit beam in response to determinations of the size of theviewer's pupil. For example, as the pupil size decreases, the size ofthe exit beam may also decrease. In some embodiments, the exit beam sizemay be varied using a variable aperture.

The wearable system 400 can include an outward-facing imaging system 464(e.g., a digital camera) that images a portion of the world 470. Thisportion of the world 470 may be referred to as the field of view (FOV)of a world camera and the imaging system 464 is sometimes referred to asan FOV camera. The FOV of the world camera may or may not be the same asthe FOV of a viewer 210 which encompasses a portion of the world 470 theviewer 210 perceives at a given time. For example, in some situations,the FOV of the world camera may be larger than the viewer 210 of theviewer 210 of the wearable system 400. The entire region available forviewing or imaging by a viewer may be referred to as the field of regard(FOR). The FOR may include 4π steradians of solid angle surrounding thewearable system 400 because the wearer can move his body, head, or eyesto perceive substantially any direction in space. In other contexts, thewearer's movements may be more constricted, and accordingly the wearer'sFOR may subtend a smaller solid angle. Images obtained from theoutward-facing imaging system 464 can be used to track gestures made bythe user (e.g., hand or finger gestures), detect objects in the world470 in front of the user, and so forth.

The wearable system 400 can include an audio sensor 232, e.g., amicrophone, to capture ambient sound. As described above, in someembodiments, one or more other audio sensors can be positioned toprovide stereo sound reception useful to the determination of locationof a speech source. The audio sensor 232 can comprise a directionalmicrophone, as another example, which can also provide such usefuldirectional information as to where the audio source is located. Thewearable system 400 can use information from both the outward-facingimaging system 464 and the audio sensor 230 in locating a source ofspeech, or to determine an active speaker at a particular moment intime, etc. For example, the wearable system 400 can use the voicerecognition alone or in combination with a reflected image of thespeaker (e.g., as seen in a mirror) to determine the identity of thespeaker. As another example, the wearable system 400 can determine aposition of the speaker in an environment based on sound acquired fromdirectional microphones. The wearable system 400 can parse the soundcoming from the speaker's position with speech recognition algorithms todetermine the content of the speech and use voice recognition techniquesto determine the identity (e.g., name or other demographic information)of the speaker.

The wearable system 400 can also include an inward-facing imaging system466 (e.g., a digital camera), which observes the movements of the user,such as the eye movements and the facial movements. The inward-facingimaging system 466 may be used to capture images of the eye 410 todetermine the size and/or orientation of the pupil of the eye 304. Theinward-facing imaging system 466 can be used to obtain images for use indetermining the direction the user is looking (e.g., eye pose) or forbiometric identification of the user (e.g., via iris identification). Insome embodiments, at least one camera may be utilized for each eye, toseparately determine the pupil size or eye pose of each eyeindependently, thereby allowing the presentation of image information toeach eye to be dynamically tailored to that eye. In some otherembodiments, the pupil diameter or orientation of only a single eye 410(e.g., using only a single camera per pair of eyes) is determined andassumed to be similar for both eyes of the user. The images obtained bythe inward-facing imaging system 466 may be analyzed to determine theuser's eye pose or mood, which can be used by the wearable system 400 todecide which audio or visual content should be presented to the user.The wearable system 400 may also determine head pose (e.g., headposition or head orientation) using sensors such as IMUs,accelerometers, gyroscopes, etc.

The wearable system 400 can include a user input device 466 by which theuser can input commands to the controller 460 to interact with thewearable system 400. For example, the user input device 466 can includea trackpad, a touchscreen, a joystick, a multiple degree-of-freedom(DOF) controller, a capacitive sensing device, a game controller, akeyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, atotem (e.g., functioning as a virtual user input device), and so forth.A multi-DOF controller can sense user input in some or all possibletranslations (e.g., left/right, forward/backward, or up/down) orrotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOFcontroller which supports the translation movements may be referred toas a 3DOF while a multi-DOF controller which supports the translationsand rotations may be referred to as 6DOF. In some cases, the user mayuse a finger (e.g., a thumb) to press or swipe on a touch-sensitiveinput device to provide input to the wearable system 400 (e.g., toprovide user input to a user interface provided by the wearable system400). The user input device 466 may be held by the user's hand duringthe use of the wearable system 400. The user input device 466 can be inwired or wireless communication with the wearable system 400.

FIG. 5 shows an example of exit beams outputted by a waveguide. Onewaveguide is illustrated, but it will be appreciated that otherwaveguides in the waveguide assembly 480 may function similarly, wherethe waveguide assembly 480 includes multiple waveguides. Light 520 isinjected into the waveguide 432 b at the input edge 432 c of thewaveguide 432 b and propagates within the waveguide 432 b by TIR. Atpoints where the light 520 impinges on the DOE 432 a, a portion of thelight exits the waveguide as exit beams 510. The exit beams 510 areillustrated as substantially parallel but they may also be redirected topropagate to the eye 410 at an angle (e.g., forming divergent exitbeams), depending on the depth plane associated with the waveguide 432b. It will be appreciated that substantially parallel exit beams may beindicative of a waveguide with light extracting optical elements thatoutcouple light to form images that appear to be set on a depth plane ata large distance (e.g., optical infinity) from the eye 410. Otherwaveguides or other sets of light extracting optical elements may outputan exit beam pattern that is more divergent, which would require the eye410 to accommodate to a closer distance to bring it into focus on theretina and would be interpreted by the brain as light from a distancecloser to the eye 410 than optical infinity.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem, usedin the generation of a multi-focal volumetric display, image, or lightfield. The optical system can include a waveguide apparatus, an opticalcoupler subsystem to optically couple light to or from the waveguideapparatus, and a control subsystem. The optical system can be used togenerate a multi-focal volumetric, image, or light field. The opticalsystem can include one or more primary planar waveguides 632 a (only oneis shown in FIG. 6) and one or more DOEs 632 b associated with each ofat least some of the primary waveguides 632 a. The planar waveguides 632b can be similar to the waveguides 432 b, 434 b, 436 b, 438 b, 440 bdiscussed with reference to FIG. 4. The optical system may employ adistribution waveguide apparatus to relay light along a first axis(vertical or Y-axis in view of FIG. 6), and expand the light's effectiveexit pupil along the first axis (e.g., Y-axis). The distributionwaveguide apparatus may, for example, include a distribution planarwaveguide 622 b and at least one DOE 622 a (illustrated by doubledash-dot line) associated with the distribution planar waveguide 622 b.The distribution planar waveguide 622 b may be similar or identical inat least some respects to the primary planar waveguide 632 b, having adifferent orientation therefrom. Likewise, at least one DOE 622 a may besimilar to or identical in at least some respects to the DOE 632 a. Forexample, the distribution planar waveguide 622 b or DOE 622 a may becomprised of the same materials as the primary planar waveguide 632 b orDOE 632 a, respectively. Embodiments of the optical display system 600shown in FIG. 6 can be integrated into the wearable system 200 shown inFIG. 2.

The relayed and exit-pupil expanded light may be optically coupled fromthe distribution waveguide apparatus into the one or more primary planarwaveguides 632 b. The primary planar waveguide 632 b can relay lightalong a second axis, preferably orthogonal to first axis (e.g.,horizontal or X-axis in view of FIG. 6). Notably, the second axis can bea non-orthogonal axis to the first axis. The primary planar waveguide632 b expands the light's effective exit pupil along that second axis(e.g., X-axis). For example, the distribution planar waveguide 622 b canrelay and expand light along the vertical or Y-axis, and pass that lightto the primary planar waveguide 632 b which can relay and expand lightalong the horizontal or X-axis.

The optical system may include one or more sources of colored light(e.g., red, green, and blue laser light) 610 which may be opticallycoupled into a proximal end of a single mode optical fiber 640. A distalend of the optical fiber 640 may be threaded or received through ahollow tube 642 of piezoelectric material. The distal end protrudes fromthe tube 642 as fixed-free flexible cantilever 644. The piezoelectrictube 642 can be associated with four quadrant electrodes (notillustrated). The electrodes may, for example, be plated on the outside,outer surface or outer periphery or diameter of the tube 642. A coreelectrode (not illustrated) may also be located in a core, center, innerperiphery or inner diameter of the tube 642.

Drive electronics 650, for example electrically coupled via wires 660,drive opposing pairs of electrodes to bend the piezoelectric tube 642 intwo axes independently. The protruding distal tip of the optical fiber644 has mechanical modes of resonance. The frequencies of resonance candepend upon a diameter, length, and material properties of the opticalfiber 644. By vibrating the piezoelectric tube 642 near a first mode ofmechanical resonance of the fiber cantilever 644, the fiber cantilever644 can be caused to vibrate, and can sweep through large deflections.

By stimulating resonant vibration in two axes, the tip of the fibercantilever 644 is scanned biaxially in an area filling two-dimensional(2D) scan. By modulating an intensity of light source(s) 610 insynchrony with the scan of the fiber cantilever 644, light emerging fromthe fiber cantilever 644 can form an image. Descriptions of such a setup are provided in U.S. Patent Publication No. 2014/0003762, which isincorporated by reference herein in its entirety.

A component of an optical coupler subsystem can collimate the lightemerging from the scanning fiber cantilever 644. The collimated lightcan be reflected by mirrored surface 648 into the narrow distributionplanar waveguide 622 b which contains the at least one diffractiveoptical element (DOE) 622 a. The collimated light can propagatevertically (relative to the view of FIG. 6) along the distributionplanar waveguide 622 b by TIR, and in doing so repeatedly intersectswith the DOE 622 a. The DOE 622 a preferably has a low diffractionefficiency. This can cause a fraction (e.g., 10%) of the light to bediffracted toward an edge of the larger primary planar waveguide 632 bat each point of intersection with the DOE 622 a, and a fraction of thelight to continue on its original trajectory down the length of thedistribution planar waveguide 622 b via TIR.

At each point of intersection with the DOE 622 a, additional light canbe diffracted toward the entrance of the primary waveguide 632 b. Bydividing the incoming light into multiple outcoupled sets, the exitpupil of the light can be expanded vertically by the DOE 622 a in thedistribution planar waveguide 622 b. This vertically expanded lightcoupled out of distribution planar waveguide 622 b can enter the edge ofthe primary planar waveguide 632 b.

Light entering primary waveguide 632 b can propagate horizontally(relative to the view of FIG. 6) along the primary waveguide 632 b viaTIR. As the light intersects with DOE 632 a at multiple points as itpropagates horizontally along at least a portion of the length of theprimary waveguide 632 b via TIR. The DOE 632 a may advantageously bedesigned or configured to have a phase profile that is a summation of alinear diffraction pattern and a radially symmetric diffractive pattern,to produce both deflection and focusing of the light. The DOE 632 a mayadvantageously have a low diffraction efficiency (e.g., 10%), so thatonly a portion of the light of the beam is deflected toward the eye ofthe view with each intersection of the DOE 632 a while the rest of thelight continues to propagate through the primary waveguide 632 b viaTIR.

At each point of intersection between the propagating light and the DOE632 a, a fraction of the light is diffracted toward the adjacent face ofthe primary waveguide 632 b allowing the light to escape the TIR, andemerge from the face of the primary waveguide 632 b. In someembodiments, the radially symmetric diffraction pattern of the DOE 632 aadditionally imparts a focus level to the diffracted light, both shapingthe light wavefront (e.g., imparting a curvature) of the individual beamas well as steering the beam at an angle that matches the designed focuslevel.

Accordingly, these different pathways can cause the light to be coupledout of the primary planar waveguide 632 b by a multiplicity of DOEs 632a at different angles, focus levels, or yielding different fill patternsat the exit pupil. Different fill patterns at the exit pupil can bebeneficially used to create a light field display with multiple depthplanes. Each layer in the waveguide assembly or a set of layers (e.g., 3layers) in the stack may be employed to generate a respective color(e.g., red, blue, green). Thus, for example, a first set of threeadjacent layers may be employed to respectively produce red, blue andgreen light at a first focal depth. A second set of three adjacentlayers may be employed to respectively produce red, blue and green lightat a second focal depth. Multiple sets may be employed to generate afull 3D or 4D color image light field with various focal depths.

Other Components of the Wearable System

In many implementations, the wearable system may include othercomponents in addition or in alternative to the components of thewearable system described above. The wearable system may, for example,include one or more haptic devices or components. The haptic devices orcomponents may be operable to provide a tactile sensation to a user. Forexample, the haptic devices or components may provide a tactilesensation of pressure or texture when touching virtual content (e.g.,virtual objects, virtual tools, other virtual constructs). The tactilesensation may replicate a feel of a physical object which a virtualobject represents, or may replicate a feel of an imagined object orcharacter (e.g., a dragon) which the virtual content represents. In someimplementations, haptic devices or components may be worn by the user(e.g., a user wearable glove). In some implementations, haptic devicesor components may be held by the user.

The wearable system may, for example, include one or more physicalobjects which are manipulable by the user to allow input or interactionwith the wearable system. These physical objects may be referred toherein as totems. Some totems may take the form of inanimate objects,such as for example, a piece of metal or plastic, a wall, a surface oftable. In certain implementations, the totems may not actually have anyphysical input structures (e.g., keys, triggers, joystick, trackball,rocker switch). Instead, the totem may simply provide a physicalsurface, and the wearable system may render a user interface so as toappear to a user to be on one or more surfaces of the totem. Forexample, the wearable system may render an image of a computer keyboardand trackpad to appear to reside on one or more surfaces of a totem. Forexample, the wearable system may render a virtual computer keyboard andvirtual trackpad to appear on a surface of a thin rectangular plate ofaluminum which serves as a totem. The rectangular plate does not itselfhave any physical keys or trackpad or sensors. However, the wearablesystem may detect user manipulation or interaction or touches with therectangular plate as selections or inputs made via the virtual keyboardor virtual trackpad. The user input device 466 (shown in FIG. 4) may bean embodiment of a totem, which may include a trackpad, a touchpad, atrigger, a joystick, a trackball, a rocker or virtual switch, a mouse, akeyboard, a multi-degree-of-freedom controller, or another physicalinput device. A user may use the totem, alone or in combination withposes, to interact with the wearable system or other users.

Example Wearable Systems, Environments, and Interfaces

A wearable system may employ various mapping related techniques in orderto achieve high depth of field in the rendered light fields. In mappingout the virtual world, it is advantageous to know all the features andpoints in the real world to accurately portray virtual objects inrelation to the real world. To this end, FOV images captured from usersof the wearable system can be added to a world model by including newpictures that convey information about various points and features ofthe real world. For example, the wearable system can collect a set ofmap points (such as 2D points or 3D points) and find new map points torender a more accurate version of the world model. The world model of afirst user can be communicated (e.g., over a network such as a cloudnetwork) to a second user so that the second user can experience theworld surrounding the first user.

FIG. 7 is a block diagram of an example of an MR environment 700. The MRenvironment 700 may be configured to receive input (e.g., visual input702 from the user's wearable system, stationary input 704 such as roomcameras, sensory input 706 from various sensors, gestures, totems, eyetracking, user input from the user input device 466 etc.) from one ormore user wearable systems (e.g., wearable system 200 or display system220) or stationary room systems (e.g., room cameras, etc.). The wearablesystems can use various sensors (e.g., accelerometers, gyroscopes,temperature sensors, movement sensors, depth sensors, GPS sensors,inward-facing imaging system, outward-facing imaging system, etc.) todetermine the location and various other attributes of the environmentof the user. This information may further be supplemented withinformation from stationary cameras in the room that may provide imagesor various cues from a different point of view. The image data acquiredby the cameras (such as the room cameras and/or the cameras of theoutward-facing imaging system) may be reduced to a set of mappingpoints.

One or more object recognizers 708 can crawl through the received data(e.g., the collection of points) and recognize or map points, tagimages, attach semantic information to objects with the help of a mapdatabase 710. The map database 710 may comprise various points collectedover time and their corresponding objects. The various devices and themap database can be connected to each other through a network (e.g.,LAN, WAN, etc.) to access the cloud.

Based on this information and collection of points in the map database,the object recognizers 708 a to 708 n may recognize objects in anenvironment. For example, the object recognizers can recognize faces,persons, windows, walls, user input devices, televisions, documents(e.g., travel tickets, driver's license, passport as described in thesecurity examples herein), other objects in the user's environment, etc.One or more object recognizers may be specialized for object withcertain characteristics. For example, the object recognizer 708 a may beused to recognizer faces, while another object recognizer may be usedrecognize documents.

The object recognitions may be performed using a variety of computervision techniques. For example, the wearable system can analyze theimages acquired by the outward-facing imaging system 464 (shown in FIG.4) to perform scene reconstruction, event detection, video tracking,object recognition (e.g., persons or documents), object pose estimation,facial recognition (e.g., from a person in the environment or an imageon a document), learning, indexing, motion estimation, or image analysis(e.g., identifying indicia within documents such as photos, signatures,identification information, travel information, etc.), and so forth. Oneor more computer vision algorithms may be used to perform these tasks.Non-limiting examples of computer vision algorithms include:Scale-invariant feature transform (SIFT), speeded up robust features(SURF), oriented FAST and rotated BRIEF (ORB), binary robust invariantscalable keypoints (BRISK), fast retina keypoint (FREAK), Viola-Jonesalgorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunkalgorithm, Mean-shift algorithm, visual simultaneous location andmapping (vSLAM) techniques, a sequential Bayesian estimator (e.g.,Kalman filter, extended Kalman filter, etc.), bundle adjustment,Adaptive thresholding (and other thresholding techniques), IterativeClosest Point (ICP), Semi Global Matching (SGM), Semi Global BlockMatching (SGBM), Feature Point Histograms, various machine learningalgorithms (such as e.g., support vector machine, k-nearest neighborsalgorithm, Naive Bayes, neural network (including convolutional or deepneural networks), or other supervised/unsupervised models, etc.), and soforth.

The object recognitions can additionally or alternatively be performedby a variety of machine learning algorithms. Once trained, the machinelearning algorithm can be stored by the HMD. Some examples of machinelearning algorithms can include supervised or non-supervised machinelearning algorithms, including regression algorithms (such as, forexample, Ordinary Least Squares Regression), instance-based algorithms(such as, for example, Learning Vector Quantization), decision treealgorithms (such as, for example, classification and regression trees),Bayesian algorithms (such as, for example, Naive Bayes), clusteringalgorithms (such as, for example, k-means clustering), association rulelearning algorithms (such as, for example, a-priori algorithms),artificial neural network algorithms (such as, for example, Perceptron),deep learning algorithms (such as, for example, Deep Boltzmann Machine,or deep neural network), dimensionality reduction algorithms (such as,for example, Principal Component Analysis), ensemble algorithms (suchas, for example, Stacked Generalization), and/or other machine learningalgorithms. In some embodiments, individual models can be customized forindividual data sets. For example, the wearable device can generate orstore a base model. The base model may be used as a starting point togenerate additional models specific to a data type (e.g., a particularuser in the telepresence session), a data set (e.g., a set of additionalimages obtained of the user in the telepresence session), conditionalsituations, or other variations. In some embodiments, the wearable HMDcan be configured to utilize a plurality of techniques to generatemodels for analysis of the aggregated data. Other techniques may includeusing pre-defined thresholds or data values.

Based on this information and collection of points in the map database,the object recognizers 708 a to 708 n may recognize objects andsupplement objects with semantic information to give life to theobjects. For example, if the object recognizer recognizes a set ofpoints to be a door, the system may attach some semantic information(e.g., the door has a hinge and has a 90 degree movement about thehinge). If the object recognizer recognizes a set of points to be amirror, the system may attach semantic information that the mirror has areflective surface that can reflect images of objects in the room. Thesemantic information can include affordances of the objects as describedherein. For example, the semantic information may include a normal ofthe object. The system can assign a vector whose direction indicates thenormal of the object. Over time the map database grows as the system(which may reside locally or may be accessible through a wirelessnetwork) accumulates more data from the world. Once the objects arerecognized, the information may be transmitted to one or more wearablesystems. For example, the MR environment 700 may include informationabout a scene happening in California. The environment 700 may betransmitted to one or more users in New York. Based on data receivedfrom an FOV camera and other inputs, the object recognizers and othersoftware components can map the points collected from the variousimages, recognize objects etc., such that the scene may be accurately“passed over” to a second user, who may be in a different part of theworld. The environment 700 may also use a topological map forlocalization purposes.

FIG. 8 is a process flow diagram of an example of a method 800 ofrendering virtual content in relation to recognized objects. The method800 describes how a virtual scene may be presented to a user of thewearable system. The user may be geographically remote from the scene.For example, the user may be in New York, but may want to view a scenethat is presently going on in California, or may want to go on a walkwith a friend who resides in California.

At block 810, the wearable system may receive input from the user andother users regarding the environment of the user. This may be achievedthrough various input devices, and knowledge already possessed in themap database. The user's FOV camera, sensors, GPS, eye tracking, etc.,convey information to the system at block 810. The system may determinesparse points based on this information at block 820. The sparse pointsmay be used in determining pose data (e.g., head pose, eye pose, bodypose, or hand gestures) that can be used in displaying and understandingthe orientation and position of various objects in the user'ssurroundings. The object recognizers 708 a-708 n may crawl through thesecollected points and recognize one or more objects using a map databaseat block 830. This information may then be conveyed to the user'sindividual wearable system at block 840, and the desired virtual scenemay be accordingly displayed to the user at block 850. For example, thedesired virtual scene (e.g., user in CA) may be displayed at theappropriate orientation, position, etc., in relation to the variousobjects and other surroundings of the user in New York.

FIG. 9 is a block diagram of another example of a wearable system. Inthis example, the wearable system 900 comprises a map 920, which mayinclude the map database 710 containing map data for the world. The mapmay partly reside locally on the wearable system, and may partly resideat networked storage locations accessible by wired or wireless network(e.g., in a cloud system). A pose process 910 may be executed on thewearable computing architecture (e.g., processing module 260 orcontroller 460) and utilize data from the map 920 to determine positionand orientation of the wearable computing hardware or user. Pose datamay be computed from data collected on the fly as the user isexperiencing the system and operating in the world. The data maycomprise images, data from sensors (such as inertial measurement units,which generally comprise accelerometer and gyroscope components) andsurface information pertinent to objects in the real or virtualenvironment.

A sparse point representation may be the output of a simultaneouslocalization and mapping (e.g., SLAM or vSLAM, referring to aconfiguration wherein the input is images/visual only) process. Thesystem can be configured to not only find out where in the world thevarious components are, but what the world is made of. Pose may be abuilding block that achieves many goals, including populating the mapand using the data from the map.

In one embodiment, a sparse point position may not be completelyadequate on its own, and further information may be needed to produce amultifocal AR, VR, or MR experience. Dense representations, generallyreferring to depth map information, may be utilized to fill this gap atleast in part. Such information may be computed from a process referredto as Stereo 940, wherein depth information is determined using atechnique such as triangulation or time-of-flight sensing. Imageinformation and active patterns (such as infrared patterns created usingactive projectors), images acquired from image cameras, or handgestures/totem 950 may serve as input to the Stereo process 940. Asignificant amount of depth map information may be fused together, andsome of this may be summarized with a surface representation. Forexample, mathematically definable surfaces may be efficient (e.g.,relative to a large point cloud) and digestible inputs to otherprocessing devices like game engines. Thus, the output of the stereoprocess (e.g., a depth map) 940 may be combined in the fusion process930. Pose 910 may be an input to this fusion process 930 as well, andthe output of fusion 930 becomes an input to populating the map process920. Sub-surfaces may connect with each other, such as in topographicalmapping, to form larger surfaces, and the map becomes a large hybrid ofpoints and surfaces.

To resolve various aspects in a mixed reality process 960, variousinputs may be utilized. For example, in the embodiment depicted in FIG.9, Game parameters may be inputs to determine that the user of thesystem is playing a monster battling game with one or more monsters atvarious locations, monsters dying or running away under variousconditions (such as if the user shoots the monster), walls or otherobjects at various locations, and the like. The world map may includeinformation regarding the location of the objects or semanticinformation of the objects and the world map can be another valuableinput to mixed reality. Pose relative to the world becomes an input aswell and plays a key role to almost any interactive system.

Controls or inputs from the user are another input to the wearablesystem 900. As described herein, user inputs can include visual input,gestures, totems, audio input, sensory input, etc. In order to movearound or play a game, for example, the user may need to instruct thewearable system 900 regarding what he or she wants to do. Beyond justmoving oneself in space, there are various forms of user controls thatmay be utilized. In one embodiment, a totem (e.g. a user input device),or an object such as a toy gun may be held by the user and tracked bythe system. The system preferably will be configured to know that theuser is holding the item and understand what kind of interaction theuser is having with the item (e.g., if the totem or object is a gun, thesystem may be configured to understand location and orientation, as wellas whether the user is clicking a trigger or other sensed button orelement which may be equipped with a sensor, such as an IMU, which mayassist in determining what is going on, even when such activity is notwithin the field of view of any of the cameras.)

Hand gesture tracking or recognition may also provide input information.The wearable system 900 may be configured to track and interpret handgestures for button presses, for gesturing left or right, stop, grab,hold, etc. For example, in one configuration, the user may want to flipthrough emails or a calendar in a non-gaming environment, or do a “fistbump” with another person or player. The wearable system 900 may beconfigured to leverage a minimum amount of hand gesture, which may ormay not be dynamic. For example, the gestures may be simple staticgestures like open hand for stop, thumbs up for ok, thumbs down for notok; or a hand flip right, or left, or up/down for directional commands.

Eye tracking is another input (e.g., tracking where the user is lookingto control the display technology to render at a specific depth orrange). In one embodiment, vergence of the eyes may be determined usingtriangulation, and then using a vergence/accommodation model developedfor that particular person, accommodation may be determined. Eyetracking can be performed by the eye camera(s) to determine eye gaze(e.g., direction or orientation of one or both eyes). Other techniquescan be used for eye tracking such as, e.g., measurement of electricalpotentials by electrodes placed near the eye(s) (e.g.,electrooculography).

Speech tracking can be another input can be used alone or in combinationwith other inputs (e.g., totem tracking, eye tracking, gesture tracking,etc.). Speech tracking may include speech recognition, voicerecognition, alone or in combination. The system 900 can include anaudio sensor (e.g., a microphone) that receives an audio stream from theenvironment. The system 900 can incorporate voice recognition technologyto determine who is speaking (e.g., whether the speech is from thewearer of the ARD or another person or voice (e.g., a recorded voicetransmitted by a loudspeaker in the environment)) as well as speechrecognition technology to determine what is being said. The local data &processing module 260 or the remote processing module 270 can processthe audio data from the microphone (or audio data in another stream suchas, e.g., a video stream being watched by the user) to identify contentof the speech by applying various speech recognition algorithms, suchas, e.g., hidden Markov models, dynamic time warping (DTW)-based speechrecognitions, neural networks, deep learning algorithms such as deepfeedforward and recurrent neural networks, end-to-end automatic speechrecognitions, machine learning algorithms (described with reference toFIG. 7), or other algorithms that uses acoustic modeling or languagemodeling, etc.

The local data & processing module 260 or the remote processing module270 can also apply voice recognition algorithms which can identify theidentity of the speaker, such as whether the speaker is the user 210 ofthe wearable system 900 or another person with whom the user isconversing. Some example voice recognition algorithms can includefrequency estimation, hidden Markov models, Gaussian mixture models,pattern matching algorithms, neural networks, matrix representation,Vector Quantization, speaker diarisation, decision trees, and dynamictime warping (DTW) technique. Voice recognition techniques can alsoinclude anti-speaker techniques, such as cohort models, and worldmodels. Spectral features may be used in representing speakercharacteristics. The local data & processing module or the remote dataprocessing module 270 can use various machine learning algorithmsdescribed with reference to FIG. 7 to perform the voice recognition.

With regard to the camera systems, the example wearable system 900 shownin FIG. 9 can include three pairs of cameras: a relative wide FOV orpassive SLAM pair of cameras arranged to the sides of the user's face, adifferent pair of cameras oriented in front of the user to handle thestereo imaging process 940 and also to capture hand gestures andtotem/object tracking in front of the user's face. The FOV cameras andthe pair of cameras for the stereo process 940 may be a part of theoutward-facing imaging system 464 (shown in FIG. 4). The wearable system900 can include eye tracking cameras (which may be a part of aninward-facing imaging system 462 shown in FIG. 4) oriented toward theeyes of the user in order to triangulate eye vectors and otherinformation. The wearable system 900 may also comprise one or moretextured light projectors (such as infrared (IR) projectors) to injecttexture into a scene.

FIG. 10 is a process flow diagram of an example of a method 1000 forinteracting with a virtual user interface. The method 1000 may beperformed by the wearable system described herein. The method 1000 mayperform the method 1000 in a telepresence session.

At block 1010, the wearable system may identify a particular UI. Thetype of UI may be predetermined by the user. The wearable system mayidentify that a particular UI needs to be populated based on a userinput (e.g., gesture, visual data, audio data, sensory data, directcommand, etc.). The UI may be specific to a telepresence session. Atblock 1020, the wearable system may generate data for the virtual UI.For example, data associated with the confines, general structure, shapeof the UI etc., may be generated. In addition, the wearable system maydetermine map coordinates of the user's physical location so that thewearable system can display the UI in relation to the user's physicallocation. For example, if the UI is body centric, the wearable systemmay determine the coordinates of the user's physical stance, head pose,or eye pose such that a ring UI can be displayed around the user or aplanar UI can be displayed on a wall or in front of the user. In thetelepresence context, the UI may be displayed as if the UI weresurrounding user to create a tangible sense of another user's presencein the environment (e.g., the UI can display virtual avatars of theparticipants around the user). If the UI is hand centric, the mapcoordinates of the user's hands may be determined. These map points maybe derived through data received through the FOV cameras, sensory input,or any other type of collected data.

At block 1030, the wearable system may send the data to the display fromthe cloud or the data may be sent from a local database to the displaycomponents. At block 1040, the UI is displayed to the user based on thesent data. For example, a light field display can project the virtual UIinto one or both of the user's eyes. Once the virtual UI has beencreated, the wearable system may simply wait for a command from the userto generate more virtual content on the virtual UI at block 1050. Forexample, the UI may be a body centric ring around the user's body or thebody of a person in the user's environment (e.g., a traveler). Thewearable system may then wait for the command (a gesture, a head or eyemovement, voice command, input from a user input device, etc.), and ifit is recognized (block 1060), virtual content associated with thecommand may be displayed to the user (block 1070).

Example of Wearable Devices for Generating a Face Model

FIG. 11 illustrates an example wearable device which can acquire imagesof the user's face while the user is putting on the wearable device. Theimages acquired while the user is putting on (or taking off) thewearable device may be used to generate a face model of the user. Thewearable device 1150 can be an example head-mounted device (HMD)described with reference to FIG. 2. The wearable device 1150 can includean imaging system 1160 which is configured to image the user's 210 face.For example, the imaging system 1160 may include sensors such as eyecameras (e.g., eye camera 1160 a and eye camera 1160 b) configured toimage the periocular region of the user's eyes 1110 while the user 210is wearing the wearable device. In this example, the eye 1110 b cancorrespond to the eye 302 and the eye 1110 a can correspond to the eye304 shown in FIG. 3. In some implementations, the imaging system 1160may be an embodiment of the inward-facing imaging system 462 shown inFIG. 4.

As shown in FIG. 11, the imaging system 1160 points toward the head ofthe user 210. The eye camera 1160 a may be configured to image the eye1160 a while the eye camera 1160 b may be configured to image the eye1110 b. In this figure, the optical axis 1140 a of the eye camera 1160 ais parallel to the optical axis 1140 b of the eye camera 1160 b. In someimplementations, one or both of the eye cameras may be rotated such thatthe optical axes of the two eye cameras are no longer in parallel. Forexample, the two eye cameras may point slightly towards each other(e.g., particularly if the eye cameras are disposed near outside edgesof the frame of the device 1150). This implementation may beadvantageous because it can create a cross eyed configuration which canincrease the overlap of the field of view (FOV) between the two camerasas well as to allow the two eye cameras to image the face at a closerdistance.

Each eye camera may have a FOV. For example, the FOV for the eye camera1160 a can include the region 1120 a and the region 1130. The FOV forthe eye camera 1160 b can include the region 1120 b and the region 1130.The FOV of the eye camera 1160 a and the FOV of the eye camera 1160 bmay overlap at the region 1130. Because of this overlapping FOV 1130, insome embodiments, the two eye cameras may be treated as a singlestereoscopic imaging system. The two eye cameras may take images of theface when the face is within the overlapping FOV in order to provide a3D image of the user's face.

In some situations, when the wearable device 1150 is too close to theuser 210, the eye cameras may be out of focus. For example, assuming theperiocular separation for the user is 46 mm (typical for an adult male)and each of the two eye cameras has a horizontal FOV of 66 degrees(appropriate for eye-tracking), then the wearable device may takepictures when the distance between the face and the wearable device isat least about 175 mm. The minimum focal distance for the lenses of manyeye cameras is approximately 14 mm. If the lenses have fixed focallength, their depth of focus needs to be about 65 diopters.

If the images are obtained when there is insufficient depth of focus,the wearable device 1150 may treat the images as low resolution images.As a result, the face model generated by the wearable device may have alower fidelity or have sparse representations of gross facial features.Such face model may still be used to deduce an interocular separationfor the user, which is useful for determining whether the wearabledevice fits the user's face.

Example Triggers for Imaging the User's Face

The wearable device 1150 can use a variety of techniques to determinethe triggers for starting and stopping imaging the user 210. Forexample, the wearable device 1150 may be configured to start imaging theuser's face when it detects that the user is putting on (or taking off)the wearable device 1150. Advantageously, the triggers for initiating orstopping image acquisition can be based on data related to the movementof the wearable device 1150 (e.g., where such movement may be measuredusing an IMU in the device) or images acquired by one or more cameras ofthe wearable device 1150 (e.g., cameras in the inward-facing imagingsystem 462 or the outward-facing imaging system 464, which detect, forexample, regions of the user's face getting larger or smaller as thedevice gets closer or farther away from the user's face). Thus, thewearable device can automatically initiate or stop the image acquisitionfree from user interventions.

The wearable device 1150 can use various sensors described withreference to FIGS. 2 and 7 for the detection of movement of the device1150. The example sensors 1170 a, 1170 b (shown in FIG. 11) are disposedon the frame of the device 1150 (e.g., on the ear stems). The sensors1170 a, 1170 b can comprise inertial measurement units, pressuresensors, proximity sensors, etc. In other implementations, sensors aredisposed on only one side of the device 1150 (e.g., on one ear stem).The data acquired by the sensors may be analyzed against a correspondingthreshold level (e.g., threshold acceleration, threshold pressure,threshold proximity). If the data pass the threshold level, the wearabledevice 1150 may start or stop the imaging process.

As an example, when a user lifts up the wearable device 1150, theinertial measurement unit of the wearable device 1150 may acquire dataon the acceleration of the wearable device 1150. If the wearable device1150 determines that the acceleration exceeds certain thresholdacceleration, the wearable device 1150 may begin to image the user'sface. Once the user puts the wearable device, for example, on the head,the acceleration typically will decrease. If the wearable device 1150determines that the acceleration has reduced to a certain threshold, thewearable device 1150 may stop taking images of the user's face. Thedevice 1150 may also image the user's face when the user takes thedevice off his or her face. The device may start imaging when theacceleration passes a typical value for device removal and may continueimaging for a time period or until the device 1150 is at or beyond acertain distance away from the user's face.

As another example, the wearable device 1150 may have a pressure sensor.The pressure sensor may be located at the temple (such as the earpieces)of glasses, or the nose pad of a wearable device. When the wearabledevice 1150 is put onto the user's face, the pressure sensor may send asignal indicating that the wearable device 1150 is on the user. As aresult, the wearable device 1150 may stop acquiring images of the user'sface.

Triggers can also be based on data acquired by one or more imagingsystem of the wearable device 1150. For example, the wearable device1150 can use images obtained by the inward-facing imaging system 462 todetermine whether to stop imaging the user's face. For example, as theuser is putting on the device, the content in the images acquired by theinward-facing imaging system 462 may change. When the device is sittingon the user's head, however, the content of the images will not changeas much compared to when the user is putting on (or taking off) thedevice. Thus, the wearable device can stop recording when it observesthat a certain threshold number (e.g., 3, 5, 10, etc.) of consecutiveimage frames or images within a certain threshold duration of time havesubstantially the same content (e.g., the wearable device can stopimaging once the wearable device detects that the user's eyes appear inthe acquired images for 5 seconds consecutively). As another example, asthe user is taking off the wearable device, the inward-facing imagingsystem may initially observe an eye, then the periocular region, thenthe upper face, then the lower face, and then the user's neck. Thissequence of images would be reversed if the user were putting on thedevice. By detecting this sequence of images, the device can infer it isbeing put on (or taken off) the user's face. In some cases, the image ofthe user may become smaller than a threshold (e.g., when the device isat arm's length from the user) or may disappear completely (e.g.,because the device has been placed on a table and the imaging system nolonger points toward the user). Once the wearable device detects thatthe device is no longer on the user (e.g., by detecting the imagingsequences described above, or because the user's face does not appear inor is smaller than a threshold)), the wearable device can stop acquiringimages.

In some situations, the wearable device can continuously acquire imagesbefore the detection of the starting trigger or after the detection ofthe stopping trigger. But the wearable device can be configured toassociate the images with the generation of the face model if the imagesare acquired in-between the starting trigger and the stopping trigger.As one example, the wearable device, can detect a starting trigger basedon data acquired from IMU (e.g., where an increase in acceleration isdetected). Thus, the images acquired after this starting trigger may bestored or tagged as being associated with generation of the face model.However, when the wearable device detects the stopping trigger (e.g.,when, there is no longer acceleration or the images contain mostlyperiocular region), the wearable device will stop associating theacquired images with the generation of the face model.

The wearable device 1150 can also include sensors for measuring thedistance between the wearable device 1150 and the user 210. For example,the sensors may emit and receive signals such as acoustic or opticalsignals, and use the signals or the feedback of the signal to measurethe distance. The wearable device 1150 may also determine the distanceby analyzing images acquired by the imaging system 1160. For example,the wearable device 1150 may determine the distance based on the size ofthe face in the image, where a big size may indicate a small distancewhile a small size may indicate a large distance. The wearable device1150 may image the user's face when the distance passes a threshold oris within a certain range. For example, as shown in FIG. 11, the two eyecameras of the wearable device 1130 may stereoscopically image theuser's face when the user's face is inside of the region 1130. Once thedistance between the user's face and the wearable device 1150 becomessufficiently small so that the user's face falls outside of the region1130, the wearable device 1150 may stop imaging the user's face. Asanother example, the wearable device 1150 may stop imaging the user'sface when the distance between the user 210 and the wearable device 1150is small enough to cause the images to be out of focus.

In some implementations, the device 1150 comprises one or more proximitysensors (e.g., capacitive proximity sensors) that may be disposed alongthe frames. When the user's head is approaching a proximity sensor (orbegins to move between a pair of proximity sensors), face imaging can bestarted, and when the device 1150 is on the user's face, the imaging canstop.

The device 1150 can include a light emitter 1175 configured toilluminate toward the user's face in the region 1130. When the device1150 starts imaging, the light can be turned on to provide faceillumination, and when the device 1150 stops imaging, the light can beturned off. In some implementations, the light 1175 may be part of theinward-facing imaging system 1160. For example, one or both eye cameras1160 a and 1160 b may be able to illuminate the light.

Additional Examples for Acquiring Images of the Face

In addition to or in alternative to imaging the face using the imagingsystem 1160, the wearable device 1150 can obtain images of the faceusing other techniques. For example, the wearable device 1150 mayinclude an outward-facing imaging system (see e.g., outward facingimaging system 464 described in FIG. 4) configured to image the user'senvironment while the user is wearing the wearable device. The user canpoint the cameras of the outward-facing imaging system toward the headof the user and obtain images of the face using the outward-facingimaging system.

The outward-facing imaging system can also acquire images of the facewhen the user is near a mirror. For example, the outward-facing imagingsystem can acquire the reflected images of the user while the user isstanding in front of the mirror. The wearable system can detect thepresence of the mirror and the reflected image of the user's head usingfacial recognition algorithm described with reference to FIG. 12. Afacial recognition algorithm may be used alone or in combination with aco-motion test. In a co-motion test, the wearable system analyzes themovement of the user based on data acquired by the IMU or observed viathe outward-facing imaging system and compares such movement with themovement of the reflected image as observed by the outward-facingimaging system. If these two measured movements substantially track eachother, then the device can assume they are co-moving and the reflectedimages represent the user. The wearable system can find the reflectedimages belong to the user if the facial recognition of the reflectedimages matches the user's face or if the co-motion associated with thereflected image correlates with the user's motion as observed by thewearable device. Additional examples of detecting the presence of amirror and analyzing the reflected images of the user's face are furtherdescribed in U.S. Publication No. 2017/0206691, titled “AugmentedReality Systems and Methods Utilizing Reflections”, the disclosure ofwhich is hereby incorporated by reference in its entirety.

Furthermore, although the examples described herein are with referenceto imaging the user's face while the user is putting on the wearabledevice, the imaging can also occur when the user is taking off thewearable device. For example, the wearable system may determine theuser's identity before the user puts on the wearable device or when theuser is interacting with the wearable device. The wearable system candetermine the user's identity based on the credentials inputted by theuser or by recognizing user's identity based on the user's biometricinformation, such as, e.g., iris recognition or face recognition. Thewearable system can associate the images acquired when the wearabledevice is taken off with the identity of a user before the wearabledevice is removed. The wearable system can also combine the imagesacquired while the user is putting on the wearable device with theimages acquired while the user is taking off the wearable device togenerate the face model for the user.

Examples of Generating a Face Model Using Stereo Vision Techniques

As shown in FIG. 11, the eye camera 1160 a and the eye camera 1160 b canhave an overlapping FOV 1130. Because of this overlapping FOV, the twoeye cameras may be treated as a single stereoscopic system for imagingthe user's face when the user's face is within the region 1130.

While the user's face is within the region 1130, the eye camera 1160 aand 1160 b can capture pairs of images of the user as the wearabledevice 1150 approaches the user 210. For example, a pair of images mayinclude an image taken by the eye camera 1160 a and an image taken bythe camera 1160 b at the same time. For a pair of images, the wearabledevice 1150 can analyze information of the face using a stereo visionalgorithm such as a block-matching algorithm, a semi-global matchingalgorithm, a semi-global block-matching algorithm, disparity maps,triangulation, depth maps, a neural network algorithm, a simultaneouslocation and mapping algorithm (e.g., SLAM or v-SLAM), and so on. Forexample, the wearable device may associate depths to many or all of thepixels in the images based on a comparison between the image acquired bythe camera 1160 a and the image acquired by the camera 1160 b.

The wearable device 1150 can apply the same technique to multiple pairsof images to extract information of the face. The wearable device 1150can fuse the information from the multiple pairs of images to generate aface model. The wearable device 1150 can use a variety of techniques toconsolidate the information. As an example, the wearable device 1150 mayuse a point cloud to represent the face. Clouds associated with multiplepairs of the images may be fit together using various algorithms such asan Iterative Closest Point (ICP) algorithm. The wearable device 1150 canreject outliers in the cloud data and smooth the surface of the facemodel using techniques such as clustering, averaging, or other similartechniques.

As another example, The wearable device can use keypoints to representthe face. The keypoints may be abstract keypoints such as valuesgenerated by a keypoints detector and descriptor algorithm such asscale-invariant feature transform (SIFT), speeded up robust features(SURF), oriented FAST and rotated BRIEF (ORB), and so on. The keypointsmay also be features unique to the face such as eye corners, mouthcorners, eyebrows, and so on. For each pair of images, the wearabledevice 1150 can match the keypoints in the image taken by the eye camera1160 a and the keypoints in the image taken by the eye camera 1160 b.

The wearable device 1150 can further deduce the changes of the pose(such as the position and orientation of the face) across multiple pairsof images, for example, by analyzing the position changes of thekeypoints.

The wearable device 1150 can convert the keypoints to a coordinate frameassociated with the face. Data from pairs of the images may be fusedtogether using the coordinate frame. The coordinate frame may be used toaverage, aggregate, and reject outlier data. Additionally oralternatively, the wearable device 1150 may use bundle adjustmenttechnique to generate the face model. For example, the wearable device1150 can reconstruct the face model using a single minimizationframework which accommodates all data from pairs of images as well asthe changes in the pose across pairs of images.

Examples of Generating a Face Model Using Monocular Vision Techniques

In addition to or in alternative to building a face model using stereovision techniques, the wearable device 1150 can also build the facemodel by fusing images of the face on a monocular basis. The monocularvision techniques can be advantageous when the two cameras do not havean overlapping FOV region 1130 or when the overlap is small.

For example, the camera 1160 a can take multiple monocular images as theuser is putting on the wearable device 1150. The wearable device 1150can generate a portion of the face model based on these images usingv-SLAM or similar algorithms. The wearable device 1150 can calculate atrajectory associated with the movement of the camera 1160 a based onthe keypoints in these images. Similarly, wearable device 1150 can usethe same techniques to generate another portion of the face model basedon the images taken by the eye camera 1160 and calculate the trajectoryassociated with the movement of the camera 1160 b.

Because the two cameras can be rigidly coupled to the wearable device1150, the relative position of the two cameras does not change duringthe imaging process. The wearable device can use the relative positionand angles of the two cameras and/or the trajectories to combine the twoportions of the face models into a single model. In someimplementations, the trajectories may also be used to calculateinterocular distance.

In some embodiments, the wearable device 1150 can use the images of onecamera to generate the face model even though that camera may have alimited field of view. For example, the wearable device can use imagesacquired by the eye camera 1160 a to generate a face model on a portionof the face. Because the face of the user 210 is symmetric, the wearabledevice can axially transform the portion of the face to obtain the otherportion of the face. These two portions of the face may be combinedtogether to generate the face model.

Other Example Embodiments

The images taken by the wearable device and other computing systems maybe used to generate a texture map for the face. The texture map of theface may include skin colors, eye colors, facial features such asfreckles or wrinkles, and so on. The wearable device can fuse imagestaken by the two eye cameras to generate an image of the whole face. Thefused image may be processed to enhance the quality. The wearable devicecan use techniques such as super resolution, lucky imaging, or otherimage processing techniques for increasing the quality. Additionally oralternatively, the wearable device may identify an image taken by one ofthe two eye cameras and process that image to create the texture map.For example, the wearable device may identify that an image taken by theeye camera 1160 a (shown in FIG. 11) includes the whole face of theuser. The wearable device may process that image and use that image toextract the texture map.

The face model and the texture map may be stored in the wearable deviceor in a remote storage location. They may be shared with other wearabledevices or computing systems. For example, during a telepresencesession, the face model and the texture map of a first user may beshared with the second user to create a tangible sense of the firstuser's presence in the second user's environment.

In some implementations, the face model may be generated based on imagestaken by the wearable device during multiple imaging sessions and/orbased on images acquired by other computing systems. For example, thewearable device may acquire images of the user's face while the user isputting on the wearable device and taking off the wearable device. Thewearable device may generate the face model based on images acquiredwhile the user is putting on the wearable device and images acquiredwhile the user is taking off the wearable device.

The wearable device can also update an existing face model using theacquired images. For example, the wearable device can collect new imagesof the user's face while the user is putting on the wearable device andupdate the face model previously generated for the same user based onthe new images.

The wearable device can also update a face model generic to a group ofusers using the new images. In some embodiments, people with differentdemographical information (such as age, gender, race, etc.) may havedifferent generic face models. For example, female teenagers may beassociated with a generic face model while male adults may be associatedwith another generic face model. The wearable device can select ageneric face model for the user based on the user's demographicinformation and update the generic face model with user specificinformation acquired while the user is putting on the wearable device.

The user can also customize the face model, for example, by selectingdifferent facial features and texture maps. As an example, the user canselect the appearance of a fantasy creature such as a science fictionalien during a telepresence session.

Although these examples refer to building a face model using a wearabledevice, not all processes of face model generation or updates arerequired to be performed on the wearable device. The wearable device cancommunicate with a remote computing device to generate a face model. Forexample, the wearable device can acquire images of the user's face andpass the images (alone or in combination with other information of theuser, such as, e.g., the user's demographic information) to a remotecomputing device (e.g., such as a server). The remote computing devicecan analyze the images and create the face model. The remote computingdevice can also pass the face model back to the wearable device of theuser or pass the face model to another user's wearable device (e.g.,during a telepresence session).

Example Processes for Generating a Face Model

FIG. 12 illustrates an example process for generating a face model. Theprocess 1200 may be performed by the wearable device 1150 described inFIG. 11. The wearable device 1150 can include a variety of sensors suchas one or more eye cameras and IMUs (described in FIGS. 2 and 7).

At block 1210, the wearable device can detect a movement of the wearabledevice. The movement may involve disposing the display device adjacentto a head of the user (either toward the user, for putting on thedevice, or away from the user, for taking off the device). For example,the wearable device can use acceleration data acquired by the IMUs anddetermine whether the acceleration exceeds a threshold acceleration. Ifthe acceleration exceeds the threshold acceleration, the wearable devicemay determine that the user is putting on (or taking off) the device.

At block 1220, the wearable device can capture the images of the user'sface. For example, one or more eye cameras may each image the user'sface while the user is putting on or taking off the wearable device. Theeye camera(s) may image the user's face through a video or multiplephotographs.

At block 1230, the wearable device can analyze the images taken by theone or more eye cameras. In some implementations using two eye cameras,when the two eye cameras are sufficiently far away from the user, thetwo eye cameras may have an overlapping FOV. Accordingly, the two eyecameras may be treated as a stereoscopic imaging system. The wearabledevice can analyze the images at different depths using a stereoscopicvision algorithm described with reference to FIG. 11. The result of theanalysis may be represented by a point cloud. The wearable device canalso analyze the images by extracting identifiable features of the faceusing a keypoints detector and descriptor algorithm. Accordingly, theface may be represented by keypoints of identifiable features.

At block 1240, the wearable device can combine the images taken atdifferent depths to generate a face model. The wearable device can alsogenerate the face model by aligning the identifiable features using acoordinate frame as described with reference to FIG. 11.

The one or more eye cameras, however, are not required to have anoverlapping FOV. Accordingly, at blocks 1230 and 1240, the wearabledevice may use a single eye camera and use monocular vision techniquesdescribed with reference to FIG. 11 to generate the face model. Forexample, the wearable device may analyze the images acquired by each eyecamera separately and combine the results of the analysis for each eyecamera to generate the face model or the device may have a single eyecamera (e.g., to track one of the user's eyes, with movement of theother eye inferred from movement of the measured eye) and use monocularvision techniques to generate the face model.

At optional block 1250, an operational parameter of the wearable devicemay be adjusted. The operational parameter may include a location of avirtual image rendered by the device, a relative position or anorientation of a light projector used to generate a virtual image (e.g.,one or more of the image injection devices 420, 422, 424, 426, 428),etc. The operational parameter may be adjusted based on an analysis ofthe images or the face model. For example, the wearable device canmeasure interocular separation based on the user's face model. Thewearable device can accordingly adjust the orientation of the lightprojectors corresponding to each eye to cause the virtual images to berendered in a suitable location for the user's eyes.

In addition to or as an alternative to adjusting operational parameters,the wearable device can also analyze the images for other purposes, suchas, e.g., to determine a fit of the wearable device on the user's head,perform user identification or authentication, or perform imageregistration or calibration. As an example of determining fit of thewearable device, the wearable device can analyze the appearance of theuser's periocular region to determine whether the wearable device istitled. Further descriptions of determining a fit of the wearable deviceare provided in U.S. Application No. 62/404,493, titled “Periocular Testfor Glasses Fit”, the disclosure of which is hereby incorporated byreference herein in its entirety.

As an example of determining a user's identity based on the images, thewearable device can analyze facial features of the user by applyingvarious facial recognition algorithms to the acquired images (e.g., faceshape, skin tone, characteristics of nose, eyes, cheeks, etc.). Someexample facial recognition algorithms include principal componentanalysis using eigenfaces, linear discriminant analysis, elastic bunchgraph matching using the Fisherface algorithm, the hidden Markov model,the multilinear subspace learning using tensor representation, and theneuronal motivated dynamic link matching, or a 3D face recognitionalgorithm. The device may also analyze the images to identify the irisand determine a biometric signature (e.g., an iris code), which isunique to each individual.

The wearable device can also perform image registration based on theimages acquired by the wearable device while the device is being put onor taken off the user's face. The resulting image obtained from theimage registration can include a portion of the user's environment(e.g., the user's room or another person near the user) in addition toor in alternative to the user's face.

FIG. 13A describes an example process of generating a face model usingstereo vision techniques. The example process 1300 can be performed bythe wearable device or a remote computing device (such as, e.g., acomputer or a server) alone or in combination.

At block 1310, the face images acquired by a wearable device may beaccessed. The face images may have been acquired concurrent with puttingon or taking off the device (see, e.g., blocks 1210 and 1220 of theprocess 1200). The face images include pairs of images taken atdifferent depths by the inward-facing imaging system 462. With referenceto FIG. 11, a pair of images can include a first image taken by the eyecamera 1160 a and a second image taken by the eye camera 1160 b. Thefirst image and the second image may be taken by their respectivecameras when the wearable device 1150 is at substantially the samedepth. The first image and the second image may also be taken by theirrespective cameras at substantially the same time. The accessed faceimages can also include images taken during multiple sessions. Forexample, some face images may have been taken a week prior to thepresent time while a user was putting on the wearable device, whileother face images may have been taken a day before the present time whenthe user was putting on the wearable device. The face images may bestored on the wearable device 1150 or in the remote data repository 280.The wearable device 1150 can communicate the face images to the remotedata repository 280 as the face images are being acquired or can uploadthe face images to the remote data repository 280 after the face imageshave been acquired.

At block 1312, a stereo vision algorithm may be applied to the accessedface images to calculate a depth image. Examples of stereo visionalgorithms include a block-matching algorithm, a semi-global matchingalgorithm, a semi-global block-matching algorithm, disparity maps,triangulation, depth maps, a neural network algorithm, a simultaneouslocation and mapping algorithm (e.g., SLAM or v-SLAM), and so on. Thedepth image may be a 3D model which contains information relating to thedistance between the face and the wearable device. For example, thestereo vision algorithm may be applied to one or more pairs of imagesand the resulting output can include depth assignments to many or allpixels in the original one or more pairs of images.

At block 1314, the face images can be fused together to produce a facemodel. Many techniques may be used for such fusion. As one example, theface may be treated as a point cloud (which would naturally result fromthe stereo computation at block 1312). Multiples of such clouds(resulting from multiple applications of the stereo vision algorithms)may be fit to one another using algorithms such as ICP. Subsequently,outliers may be rejected and the surface may be smoothed by clustering,averaging, or using another similar technique. The face model arisingfrom the point clouds calculation may be a dense model.

Faces may also be modeled as collections of keypoints (such as, e.g., aset of sparse, distinct, and visually salient features), or may bemodeled by the identification and localization of particular featuresunique to the face (e.g. eye corners, mouth corners, eyebrows, etc.). Ineither case, these features may be “fused” together with mathematicalcombinations to minimize uncertainty in the features' locations. As oneexample, the keypoints may be matched from one image frame to anotherimage frame, which can also deduce pose change (e.g., changes in theposition and orientation of the user's head). In this case, the featuresor keypoints may be converted to a common coordinate frame fixed to theface. Thereafter, like keypoints can be averaged, or similarlyaggregated, possibly including some degree of outlier rejection. Theface model may be a sparse model if keypoints techniques are used.

At the optional block 1316, the texture map may be applied to the facemodel. The texture map may be determined based on the user's faceimages. For example, the texture map may include skin tones as appearedin the face images.

At the optional block 1318, the face model may be communicated toanother wearable device. For example, while the user is in atelepresence session with another user, the face model may be used tocreate an avatar of the user and the face model may be passed to theother user's wearable device. The face model may also be communicated tothe user in some situations. The user can further manipulate the facemodel such as, e.g., by applying a hair style or changing skin color orappearance.

FIG. 13B describes an example process of generating a face model usingmonocular vision techniques. The example process 1350 can be performedby the wearable device or a remote computing device (such as, e.g., acomputer or a server) alone or in combination.

At block 1352, first face images and second face images can be accessed.The face images may have been acquired concurrent with putting on ortaking off the device (see, e.g., blocks 1210 and 1220 of the process1200). The first face images may be acquired by a first eye camera andthe second face images may be acquired by a second eye camera. The firsteye camera and the second eye camera may each be configured to image aportion of the user's face. As the user is putting on the wearabledevice, the first eye camera and the second eye camera may each beconfigured to take a series of images.

At block 1354, the first face images can be analyzed and fused togetherto create a first portion of a face model, while at block 1356, thesecond face images can be analyzed and fused together to create a secondportion of the face model. The first portion and the second portion ofthe face model can be created based on the first face images and thesecond face images, respectively, using various mapping techniques, suchas SLAM, v-SLAM, or other mapping techniques described with reference tothe object recognizers 708.

At block 1358, the first portion and the second portion of the facemodel can be combined to create a full face model. The wearable devicecan use the relative position and angles of the first and second camerasalone or in combination with the movement trajectories of the wearabledevice (as deduced from the first images and the second images) tocombine the two portions of the face model into a single model.

Although the examples are described with reference to a face model,similar techniques can also be applied to generate virtual images ofother parts of the body (alone or in combination with the face). Forexample, while the user is putting on the wearable device, some of theimages acquired by the inward-facing imaging system can include aportion of the user's torso, e.g., the user's neck or upper body (e.g.,shoulders). The wearable system can generate a face model in combinationwith a model of the user's neck or the upper body using similaralgorithms as described in FIGS. 11-13B. As another example, the usercan turn the outward-facing imaging system to face the user and scan theuser's body. The images acquired from such scan can also be used togenerate a model of the user's body. The model of the user's body canalso be used in a virtual avatar (e.g., during a telepresence session).

Additional Aspects of Face Model Capture with a Wearable Device

In a 1st aspect, an augmented reality (AR) system for generating athree-dimensional (3D) model of a face of a user, the system comprising:an augmented reality device (ARD) configured to display a 3D environmentto the user; an inward-facing imaging system comprising a first eyecamera and a second eye camera, wherein the inward-facing imaging systemis configured to image a portion of the face of the user; an inertialmeasurement unit (IMU) associated with the ARD and configured to detectmovements of the user; a computer processor associated with the ARD andprogrammed to: receive an indication of a movement from the IMU, whereinthe movement involves putting the ARD onto a head of the user; while theARD is being put onto the head of the user: receive first images of theface from the first eye camera; and receive second images of the facefrom the second eye camera; analyze the first images and the secondimages; and generate a face model of the face based at least partly onanalysis of the first images and the second images.

In a 2nd aspect, the system of aspect 1, wherein the IMU comprises oneor more of: an accelerometer, a compass, or a gyroscope.

In a 3rd aspect, the system of any one of aspects 1-2, wherein theindication of the movement comprises an increase in an acceleration ofthe ARD or a measurement of the acceleration of the ARD that passes athreshold acceleration.

In a 4th aspect, the system of any one of aspects 1-3, wherein toanalyze the first images and the second images, the computer processoris programmed to convert the first images and the second images to pointclouds in a 3D space using a stereo vision algorithm.

In a 5th aspect, the system of aspect 4, wherein the stereo visionalgorithm comprises at least one of a block-matching algorithm, asemi-global matching algorithm, a semi-global block-matching algorithm,or a neural network algorithm.

In a 6th aspect, the system of aspect 5, wherein to generate the facemodel, the computer processor is further programmed to combine the pointclouds using an iterative closest point algorithm.

In a 7th aspect, the system of any one of the aspects 1-6, wherein toanalyze the first images and the second images, the computer processoris further programmed to identify keypoints in the first image and thesecond image using a keypoints detector and descriptor algorithm.

In an 8th aspect, the system of any one of the aspects 1-7, to analyzethe first images and the second images, the computer processor isfurther programmed to: identify facial features of the face based atleast partly on the first images and the second images; and describe theidentified facial features with the points in the 3D space.

In a 9th aspect, the system of any one of aspects 7-8, wherein togenerate the face model, the computer processor is configured to combinefacial features or keypoints using a bundle adjustment algorithm.

In a 10th aspect, the system of any one of aspects 1-9, wherein toanalyze the first images and the second images and to generate the facemodel, the computer processor is programmed to: generate a first portionof the face model based at least partly on the first images; generate asecond portion of the face model based at least partly on the secondimages; and combine the first portion of the face model and the secondportion of the face model to obtain the face model.

In an 11th aspect, the system of aspect 10, wherein to analyze the firstimages and the second images is performed by a visual simultaneouslocation and mapping algorithm.

In a 12th aspect, the system of any one of the aspects 1-11, wherein thefirst images comprise first frames of a first video taken by the firsteye camera and the second images comprise second frames of the videotaken by the second eye camera.

In a 13th aspect, the system of aspect 12, wherein to generate the facemodel, the computer processor is programmed to combine the first framesof the video with the second frames of the video.

In a 14th aspect, the system of any one of aspects 1-13, the computerprocessor is further configured to generate a texture map associatedwith the face model of the face based at least partly on one or moreimages in the first images or the second images.

In a 15th aspect, the system of any one of aspects 1-14, wherein thecomputer processor is further configured to share the face model of theface with another user.

In a 16th aspect, the system of any one of aspects 1-15, wherein thefirst eye camera is configured to image a left eye of the user and thesecond eye camera is configured to image a right eye of the user.

In a 17th aspect, the system of any one of aspects 1-16, wherein thefirst eye camera and the second eye camera have an overlapping field ofview.

In an 18th aspect, a method of generating a three-dimensional (3D) modelof a face of a user, the method comprising: under control of a wearabledevice comprising computer hardware, a display device configured todisplay a 3D environment to the user, an imaging system configured toimage a portion of the face of the user, and an inertial measurementunit (IMU) configured to detect movements of the display device:detecting, by the IMU, a trigger for imaging a face of the user, whereinthe trigger comprises a movement involving disposing the display deviceadjacent to a head of the user; capturing, by the imaging system, imagesof at least a portion of a face of the user; analyzing the imagescaptured by the imaging system; and generating the face model based atleast partly on the analysis of the images.

In a 19th aspect, the method of claim 18, wherein detecting the triggercomprises: determining, by the IMU, an acceleration of the displaydevice; comparing the acceleration of the display device with athreshold acceleration; and detecting the trigger in response to acomparison that the acceleration exceeds the threshold acceleration.

In a 20th aspect, the method of any one of aspects 18-19, wherein one ormore of the images comprises a portion of a body of the user other thanthe face.

In a 21st aspect, the method of any one of aspects 18-20, wherein theimages comprises first images captured by a first eye camera of theimaging system and second images captured by a second eye camera of theimaging system.

In a 22nd aspect, the method of aspect 21, wherein analyzing the imagescomprises: converting the first images and the second images to pointclouds using a stereo vision algorithm.

In a 23rd aspect, the method of aspect 22, wherein the stereo visionalgorithm comprises at least one of a block-matching algorithm, asemi-global matching algorithm, a semi-global block-matching algorithm,or a neural network algorithm.

In a 24th aspect, the method of aspect 23, wherein generating the facemodel of the face comprises combining the point clouds using aniterative closest point algorithm.

In a 25th aspect, the method of any one of aspects 22-24, whereinanalyzing the images comprises identifying keypoints associated with theface of the user in the images, and wherein generating the face model ofthe face comprises generating the face model with the keypoints using abundle adjustment algorithm.

In a 26th aspect, the method of any one of aspects 22-25, whereinanalyzing the images comprise: analyzing the first images to generate afirst portion of the face model using a visual simultaneous location andmapping algorithm; and analyzing the second images to generate a secondportion of the face model using the visual simultaneous location andmapping algorithm.

In a 27th aspect, the method of aspect 26, wherein generating the acemodel of the face comprises: combining the first portion of the facemodel and the second portion of the face model to generate the facemodel.

In a 28th aspect, the method of any one of aspects 18-27, wherein theimages comprises frames of a video taken by the imaging system.

In a 29th aspect, the method of any one of aspects 18-28, furthercomprising: generating a texture map associated with the face modelbased at least partly on the images.

In a 30th aspect, the method of any one of aspects 18-29, whereingenerating the face model comprises: accessing a pre-existing facemodel; and updating the pre-existing face model based at least partly onthe analysis of the images.

In a 31st aspect, the method of aspect 30, wherein the pre-existing facemodel comprises at least one of the following: a generic face model or apreviously generated face model of the face of the user.

In a 32nd aspect, the method of any one of aspects 18-31, whereingenerating the face model comprising: accessing images of the facepreviously acquired by the wearable device or by another computingdevice; and generating the face model based at least partly on theanalysis of images captured by the imaging system and the accessedimages.

In a 33rd aspect, the method of any one of aspects 18-32, furthercomprising: communicating the face model to another display device; anddisplaying, by the other display device, an image associated with theface of the user based at least partly on the face model.

In a 34th aspect, a system for generating a three-dimensional (3D) modelof a face of a user, the system comprising: a head-mounted display (HMD)configured to present virtual content to a user; an inward-facingimaging system comprising at least one eye camera, wherein theinward-facing imaging system is configured to image at least a portionof the face of the user while the user is wearing the HMD; an inertialmeasurement unit (IMU) associated with the HMD and configured to detectmovements of the HMD; and a hardware processor programmed to: detect atrigger to initiate imaging of a face of the user, wherein the triggercomprises a movement detected by the IMU involving putting the HMD ontoa head of the user or taking the HMD off of the head of the user;activate, in response to detecting the trigger, the at least one eyecamera to acquire images; detect a stopping condition for stopping theimaging based on data acquired from at least one of the IMU or theinward-facing imaging system; analyze the images acquired by the atleast one eye camera with a stereo vision algorithm; and fuse the imagesto generate a face model of the user's face based at least partly on anoutput of the stereo vision algorithm.

In a 35th aspect, the system of aspect 34, wherein to detect thetrigger, the hardware processor is programmed to: determine anacceleration of the HMD; compare the acceleration of the HMD with athreshold acceleration; and detect the trigger in response to acomparison that the acceleration exceeds the threshold acceleration.

In a 36th aspect, the system of any one of aspects 34-35, wherein thestopping condition is detected when a distance between the HMD and thehead of the user passes a threshold distance.

In a 37th aspect, the system of any one of aspects 34-36, wherein thestereo vision algorithm comprises at least one of: a block-matchingalgorithm, a semi-global matching algorithm, a semi-globalblock-matching algorithm, a disparity map, a depth map, or a neuralnetwork algorithm.

In a 38th aspect, the system of any one of aspects 34-37, wherein the atleast one eye camera comprises a first eye camera and a second eyecamera, and wherein the first eye camera and the second eye camera havean overlapping field of view.

In a 39th aspect, the system of aspect 38, wherein the images comprisesa plurality of pairs of images, wherein each pair of images comprises afirst image acquired by the first eye camera and a second image acquiredby the second eye camera.

In a 40th aspect, the system of aspect 39, wherein a pair of images isanalyzed together with the stereo vision algorithm.

In a 41st aspect, the system of any one of aspects 39-40, wherein theoutput of the stereo vision algorithm comprises depth assignments topixels in the plurality of pairs of images.

In a 42nd aspect, the system of any one of aspects 39-41, wherein theuser's face is represented by a plurality of point clouds based on theanalysis of the images acquired by the first eye camera and the secondeye camera, and wherein to fuse the images to generate a face model, thehardware processor is programmed to: fit the plurality of clouds to oneanother; reject outliners in the plurality of clouds; and smooth asurface of the face model by at least one of clustering or averaging.

In a 43rd aspect, the system of aspect 42, wherein the fit the pluralityof clouds, the hardware processor is programmed to apply IterativeClosest Point algorithm to the plurality of clouds.

In a 44th aspect, the system of any one of aspects 34-43, wherein thehardware processor is further programmed to: determine a texture mapbased on the images; and apply the texture map to the face model.

In a 45th aspect, the system of any one of aspects 34-44, wherein thehardware processor is further programmed to pass the face model to awearable device.

In a 46th aspect, the system of any one of aspects 34-45, wherein toanalyze the images, the hardware processor is programmed to at least:identify keypoints in the images using a keypoints detector anddescriptor algorithm; or identify facial features from the images anddescribe the identified facial features with points in a 3D space.

In a 47th aspect, the system of aspect 46, wherein to fuse the images,the hardware processor is programmed to combine the keypoints or facialfeatures using a bundle adjustment algorithm.

In a 48th aspect, a method for generating a three-dimensional (3D) modelof a face of a user, the method comprising: receiving a request forgenerating a face model of a user; accessing images of the user's headacquired by an inward-facing imaging system of a wearable device,wherein the inward-facing imaging system comprises at least one eyecamera; identifying a plurality of pairs of images from the accessedimages; analyze the images by applying a stereo vision algorithm to theplurality of pairs of images; and fusing outputs obtained from saidanalyzing step to create a face model.

In a 49th aspect, the method of aspect 48, wherein the outputs comprisea depth map associated with the user's face, which contains informationrelating to distances between the face and the wearable device.

In a 50th aspect, the method of any one of aspects 48-49, wherein theimages are acquired as the wearable is being put on or taken off fromthe user.

In a 51st aspect, the method of any one of aspects 48-50, wherein the atleast one eye camera comprises a first eye camera and a second eyecamera, and a pair of images comprises a first image and a second imagethat are acquired at substantially the same time by the first eye cameraand the second eye camera respectively.

In a 52nd aspect, the method of any one of aspects 48-51, whereinanalyzing the images comprise converting the plurality of pairs ofimages into point clouds.

In a 53rd aspect, the method of aspect 52, wherein fusing the outputscomprises combining the point clouds using an iterative closest pointalgorithm.

Other Considerations

Each of the processes, methods, and algorithms described herein and/ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems caninclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someimplementations, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain implementations of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, animationsor video may include many frames, with each frame having millions ofpixels, and specifically programmed computer hardware is necessary toprocess the video data to provide a desired image processing task orapplication in a commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. The methods andmodules (or data) may also be transmitted as generated data signals(e.g., as part of a carrier wave or other analog or digital propagatedsignal) on a variety of computer-readable transmission mediums,including wireless-based and wired/cable-based mediums, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). The resultsof the disclosed processes or process steps may be stored, persistentlyor otherwise, in any type of non-transitory, tangible computer storageor may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities can be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto can be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe implementations described herein is for illustrative purposes andshould not be understood as requiring such separation in allimplementations. It should be understood that the described programcomponents, methods, and systems can generally be integrated together ina single computer product or packaged into multiple computer products.Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the implementations described inthis disclosure may be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate implementations also can be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. In addition, thearticles “a,” “an,” and “the” as used in this application and theappended claims are to be construed to mean “one or more” or “at leastone” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted can be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other implementations. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts. Additionally, other implementations are within the scope ofthe following claims. In some cases, the actions recited in the claimscan be performed in a different order and still achieve desirableresults.

What is claimed is:
 1. A system for generating a three-dimensional (3D)model of a face of a user, the system comprising: a head-mounted display(HMD) configured to present virtual content to a user; an inward-facingimaging system comprising a first eye camera and a second eye camera,wherein the first and second eye cameras are configured to image leftand right eyes of the user, respectively, while the user is wearing theHMD; and a hardware processor programmed to: acquire images of the faceof the user using the first and second eye cameras; based at least onsensor data from one or more inertial sensor of the HMD, detect movementof the HMD indicative of movement towards the head of the user; inresponse to said detected movement, for pairs of acquired images eachincluding a first image from the first eye camera and a second eye imagefrom the second eye camera: determine, based on one or more of theacquired images, a distance between the head of the user and the HMD;and in response to determining that the distance is within a range ofdistances, tag the pair of acquired images as usable in generating a 3Dmodel of the face of the user; identify one or more tagged pairs ofacquired images; for each of the one or more tagged pairs of acquiredimages, analyze the acquired images with a stereo vision algorithm; andbased at least on an output of the stereo vision algorithm, fuse the oneor more tagged pairs of acquired images to generate a 3D model of theface of the user.
 2. The system of claim 1, wherein the first eye cameraand the second eye camera have an overlapping field of view.
 3. Thesystem of claim 2, wherein the overlapping field of view of the firsteye camera and the second eye camera defines an imaging region, and thecertain range of distances comprise distances within the imaging region.4. The system of claim 1, wherein the hardware processor is programmedto detect, from at least one sensor associated with the HMD, a wornposition of the HMD on the user.
 5. The system of claim 4, wherein theat least one sensor comprises a proximity sensor.
 6. The system of claim5, wherein the proximity sensor comprises a capacitive proximity sensor.7. The system of claim 5, wherein data from the proximity sensorcomprises a distance between the user's head and the proximity sensor.8. The system of claim 4, wherein the at least one sensor comprises apressure sensor.
 9. The system of claim 8, wherein data from thepressure sensor comprises a pressure measurement.
 10. The system ofclaim 8, wherein the pressure sensor is disposed on a temple of the HMD.11. The system of claim 8, wherein the pressure sensor is disposed on anose pad of the HMD.
 12. The system of claim 4, wherein the at least onesensor comprises an inertial measurement unit (IMU).
 13. The system ofclaim 12, wherein data from the IMU comprises an accelerationmeasurement of the HMD.
 14. The system of claim 4, wherein the at leastone sensor comprises the first eye camera or the second eye camera. 15.The system of claim 14, wherein data from the first eye camera or thesecond eye camera comprises an image of the user's face.
 16. The systemof claim 1, wherein to determine the distance between the head of theuser and the HDM, the hardware processor is programmed to: analyze thepair of acquired images to estimate a size of the face of the user in animaging plane associated with the pair of acquired images; and determinewhether the size of the face of the user is less than a threshold size,wherein the threshold size is associated with the distance being outsideof the certain range of distances.
 17. A method for generating athree-dimensional (3D) model of a face of a user, the method comprising:receiving a request for generating a 3D face model of a user; based atleast on sensor data from one or more inertial sensor of a head-mounteddisplay (HMD), detect movement of the HMD indicative of movement towardsthe head of the user; in response to said detected movement, accessingimages acquired by an inward-facing imaging system of a wearable device,wherein the inward-facing imaging system comprises a first eye cameraand a second eye camera; tagging at least one image of the accessedimages in response to determining that the user's head position iswithin a certain range of distances, wherein the user's head position isbased on a size of the face of the user in the at least one image of theaccessed images; identifying a plurality of pairs of images from thetagged images; analyzing the tagged images by applying a stereo visionalgorithm to the plurality of pairs of images; and fusing outputsobtained from said analyzing to create the 3D model of the face of theuser.
 18. The method of claim 17, wherein the outputs comprise a depthmap associated with the face of the user, which contains informationrelating to distances between the face and the wearable device.
 19. Themethod of claim 17, wherein a pair of images comprises a first image anda second image that are acquired substantially concurrently by the firsteye camera and the second eye camera respectively.
 20. The method ofclaim 17, wherein analyzing the images comprise converting the pluralityof pairs of images into point clouds.
 21. The method of claim 20,wherein fusing the outputs comprises combining the point clouds using aniterative closest point algorithm.
 22. The system of claim 17, whereinthe certain range of distances comprise distances greater than 175 mm.23. The system of claim 17, wherein the certain range of distancescomprise distances greater than a distance where images from the firsteye camera or the second eye camera are out of focus.
 24. The method ofclaim 17, wherein a field of view of the first eye camera and a field ofview of the second eye camera overlap to define an imaging region, andthe certain range of distances comprise distances within the imagingregion.
 25. The method of claim 24, wherein the certain range ofdistances comprise distances greater than 175 mm.
 26. The method ofclaim 25, wherein the certain range of distances comprise distancesgreater than a distance where images from the first eye camera or thesecond eye camera are out of focus.